OpenAI's ChatGPT made its debut in Japanese parliamentary deliberations, with the premier fielding questions from an opposition lawmaker that were drawn up with the help of the chatbot. Kazuma Nakatani, of the Constitutional Democratic Party, said in a session of parliament Wednesday that he asked ChatGPT: "What kind of questions would you ask the prime minister if you were a member of the lower house of parliament?" He then used those responses to form questions for Prime Minister Fumio Kishida during a discussion around a draft amendment related to Covid-19 pandemic policy. Among the questions drawn up by ChatGPT were: "On the bill about Covid policy revision, do you think you have listened to the opinion of local government and health-care workers enough? And could you tell us how those people involved are responding to it?"

Supervised machine learning provides the learner with a set of input-output examples of the target task. Humans, however, can also learn to perform new tasks from instructions in natural language. Can machines learn to understand instructions as well? We present the Turking Test, which examines a model's ability to follow natural language instructions of varying complexity. These range from simple tasks, like retrieving the nth word of a sentence, to ones that require creativity, such as generating examples for SNLI and SQuAD in place of human intelligence workers ("turkers"). Despite our lenient evaluation methodology, we observe that a large pretrained language model performs poorly across all tasks. Analyzing the model's error patterns reveals that the model tends to ignore explicit instructions and often generates outputs that cannot be construed as an attempt to solve the task. While it is not yet clear whether instruction understanding can be captured by traditional language models, the sheer expressivity of instruction understanding makes it an appealing alternative to the rising few-shot inference paradigm.